Monitoring search query results relating to alert notifications

ABSTRACT

The disclosed embodiments relate to a system that generates an alert based on information extracted from search results generated by a query. During operation, the system executes the query to generate the search results. The system also obtains configuration information for the alert, wherein the configuration information identifies information associated with the search results, and also specifies a trigger condition for the alert. Next, when the trigger condition for the alert is met, the system uses the configuration information to generate a payload containing the identified information associated with the search results. The system then invokes alert-generating functionality and provides the payload as input to the alert-generating functionality. This enables the alert-generating functionality to use the information from the search results while performing one or more alert actions association with the alert.

RELATED APPLICATION

This application claims benefit as a Continuation of application Ser.No. 14/698,724, filed Apr. 28, 2015, the entire contents of theforegoing are hereby incorporated by reference as if fully set forthherein, under 35 U.S.C. § 120. The applicant(s) hereby rescind anydisclaimer of claim scope in the parent application(s) or theprosecution history thereof and advise the USPTO that the claims in thisapplication may be broader than any claim in the parent application(s).

BACKGROUND

Field of the Invention

The disclosed embodiments generally relate to query-processing systems.More specifically, the disclosed embodiments relate to aquery-processing system that facilitates generating alerts based onsearch results produced by the query-processing system.

Related Art

Modern data centers often comprise thousands of host computer systemsthat operate collectively to service requests from even larger numbersof remote clients. During operation, these data centers generatesignificant volumes of performance data and diagnostic information thatneed to be analyzed to diagnose performance and security problems. Tomonitor such large volumes of data, organizations often use event-basedsystems, such as the SPLUNK® ENTERPRISE system produced by Splunk Inc.of San Francisco, Calif., to store and process their performance dataand diagnostic information. The Splunk system can be used to processlarge volumes of data by using queries specified in Splunk's SearchProcessing Language (SPL).

The search results produced by such queries often contain importantinformation related to performance problems and security issues, and itis often necessary to take immediate action to deal with such problems.Hence, when a performance problem or security issue is detected, it isdesirable to generate an alert that triggers one or more “alert actions”to deal with such problems. For example, an alert action can includesending an email to a system administrator, or causing a firewall toblock packets received from a specific IP address. At present, this canonly be accomplished by manually writing a script (or application) toexamine the search results and generate an alert.

Hence, what is needed is a system that facilitates automaticallygenerating alerts based on search results generated by aquery-processing system.

SUMMARY

The disclosed embodiments relate to a system that generates an alertbased on information extracted from search results generated by a query.During operation, the system executes the query to generate the searchresults. The system also obtains configuration information for thealert, wherein the configuration information identifies informationassociated with the search results, and also specifies a triggercondition for the alert. Next, when the trigger condition for the alertis met, the system uses the configuration information to generate apayload containing the identified information associated with the searchresults. The system then invokes alert-generating functionality andprovides the payload as input to the alert-generating functionality.This enables the alert-generating functionality to use the informationobtained from the search results to generate the alert.

The disclosed embodiments also relate to a system that enables a user toconfigure alert actions based on search results generated by a query.During operation, the system presents an alert user interface (UI) to auser, wherein the alert UI enables the user to configure one or morealert actions to be performed based on the search results. Next, thesystem receives alert configuration information from the user throughthe alert UI, wherein the alert configuration information includestokens representing parameters associated with the query and the searchresults. Then, while generating an alert associated with the searchresults, the system performs a token substitution operation thatsubstitutes tokens in the alert configuration information withcorresponding parameters from the search results to generate a payloadthat is communicated to alert-generating functionality. This tokensubstitution allows the parameters to be used by the alert-generatingfunctionality while performing the one or more alert actions.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 presents a block diagram of an event-processing system inaccordance with the disclosed embodiments.

FIG. 2 presents a flow chart illustrating how indexers process, index,and store data received from forwarders in accordance with the disclosedembodiments.

FIG. 3 presents a flow chart illustrating how a search head and indexersperform a search query in accordance with the disclosed embodiments.

FIG. 4 presents a block diagram of a system for processing searchrequests that uses extraction rules for field values in accordance withthe disclosed embodiments.

FIG. 5 illustrates an exemplary search query received from a client andexecuted by search peers in accordance with the disclosed embodiments.

FIG. 6A illustrates a search screen in accordance with the disclosedembodiments.

FIG. 6B illustrates a data summary dialog that enables a user to selectvarious data sources in accordance with the disclosed embodiments.

FIG. 7 illustrates a system for generating an alert based on searchresults in accordance with the disclosed embodiments.

FIG. 8A illustrates an exemplary XML payload in accordance with thedisclosed embodiments.

FIG. 8B illustrates an exemplary JSON payload in accordance with thedisclosed embodiments.

FIG. 9 illustrates an exemplary alert-generation user interface (UI) inaccordance with the disclosed embodiments.

FIG. 10 illustrates an exemplary alert-action UI in accordance with thedisclosed embodiments.

FIG. 11 illustrates an exemplary alert-action UI for emails inaccordance with the disclosed embodiments.

FIG. 12A presents a flow chart illustrating how alerts are generated inaccordance with the disclosed embodiments.

FIG. 12B presents a flow chart illustrating how an alert can be manuallytriggered in accordance with the disclosed embodiments.

FIG. 12C presents a screenshot illustrating how an alert can be manuallytriggered through a UI in accordance with the disclosed embodiments.

FIG. 12D presents a screenshot illustrating another technique formanually triggering an alert through a UI in accordance with thedisclosed embodiments.

FIG. 13 presents a flow chart illustrating how a user configures analert action in accordance with the disclosed embodiments.

DETAILED DESCRIPTION

The disclosed embodiments relate to a system that facilitatesautomatically generating alerts based on search results generated by aquery-processing system. This system is described in more detail below,but first we describe the structure of an event-based framework in whichthis system operates.

1.1 System Overview

Modern data centers often comprise thousands of host computer systemsthat operate collectively to service requests from even larger numbersof remote clients. During operation, these data centers generatesignificant volumes of performance data and diagnostic information thatcan be analyzed to quickly diagnose performance problems. In order toreduce the size of this performance data, the data is typicallypre-processed prior to being stored based on anticipated data-analysisneeds. For example, pre-specified data items can be extracted from theperformance data and stored in a database to facilitate efficientretrieval and analysis at search time. However, the rest of theperformance data is not saved and is essentially discarded duringpre-processing. As storage capacity becomes progressively cheaper andmore plentiful, there are fewer incentives to discard this performancedata and many reasons to keep it.

This plentiful storage capacity is presently making it feasible to storemassive quantities of minimally processed performance data at “ingestiontime” for later retrieval and analysis at “search time.” Note thatperforming the analysis operations at search time provides greaterflexibility because it enables an analyst to search all of theperformance data, instead of searching pre-specified data items thatwere stored at ingestion time. This enables the analyst to investigatedifferent aspects of the performance data instead of being confined tothe pre-specified set of data items that was selected at ingestion time.

However, analyzing massive quantities of heterogeneous performance dataat search time can be a challenging task. A data center may generateheterogeneous performance data from thousands of different components,which can collectively generate tremendous volumes of performance datathat can be time-consuming to analyze. For example, this performancedata can include data from system logs, network packet data, sensordata, and data generated by various applications. Also, the unstructurednature of much of this performance data can pose additional challengesbecause of the difficulty of applying semantic meaning to unstructureddata, and the difficulty of indexing and querying unstructured datausing traditional database systems.

These challenges can be addressed by using an event-based system, suchas the SPLUNK® ENTERPRISE system produced by Splunk Inc. of SanFrancisco, Calif., to store and process performance data. The SPLUNK®ENTERPRISE system is the leading platform for providing real-timeoperational intelligence that enables organizations to collect, index,and harness machine-generated data from various websites, applications,servers, networks, and mobile devices that power their businesses. TheSPLUNK® ENTERPRISE system is particularly useful for analyzingunstructured performance data, which is commonly found in system logfiles. Although many of the techniques described herein are explainedwith reference to the SPLUNK® ENTERPRISE system, the techniques are alsoapplicable to other types of data server systems.

In the SPLUNK® ENTERPRISE system, performance data is stored as“events,” wherein each event comprises a collection of performance dataand/or diagnostic information that is generated by a computer system andis correlated with a specific point in time. Events can be derived from“time series data,” wherein time series data comprises a sequence ofdata points (e.g., performance measurements from a computer system) thatare associated with successive points in time and are typically spacedat uniform time intervals. Events can also be derived from “structured”or “unstructured” data. Structured data has a predefined format, whereinspecific data items with specific data formats reside at predefinedlocations in the data. For example, structured data can include dataitems stored in fields in a database table. In contrast, unstructureddata does not have a predefined format. This means that unstructureddata can comprise various data items having different data types thatcan reside at different locations. For example, when the data source isan operating system log, an event can include one or more lines from theoperating system log containing raw data that includes different typesof performance and diagnostic information associated with a specificpoint in time. Examples of data sources from which an event may bederived include, but are not limited to: web servers; applicationservers; databases; firewalls; routers; operating systems; and softwareapplications that execute on computer systems, mobile devices, andsensors. The data generated by such data sources can be produced invarious forms including, for example and without limitation, server logfiles, activity log files, configuration files, messages, network packetdata, performance measurements and sensor measurements. An eventtypically includes a timestamp that may be derived from the raw data inthe event, or may be determined through interpolation between temporallyproximate events having known timestamps.

The SPLUNK® ENTERPRISE system also facilitates using a flexible schemato specify how to extract information from the event data, wherein theflexible schema may be developed and redefined as needed. Note that aflexible schema may be applied to event data “on the fly,” when it isneeded (e.g., at search time), rather than at ingestion time of the dataas in traditional database systems. Because the schema is not applied toevent data until it is needed (e.g., at search time), it is referred toas a “late-binding schema.”

During operation, the SPLUNK® ENTERPRISE system starts with raw data,which can include unstructured data, machine data, performancemeasurements or other time-series data, such as data obtained fromweblogs, syslogs, or sensor readings. It divides this raw data into“portions,” and optionally transforms the data to produce timestampedevents. The system stores the timestamped events in a data store, andenables a user to run queries against the data store to retrieve eventsthat meet specified criteria, such as containing certain keywords orhaving specific values in defined fields. Note that the term “field”refers to a location in the event data containing a value for a specificdata item.

As noted above, the SPLUNK® ENTERPRISE system facilitates using alate-binding schema while performing queries on events. A late-bindingschema specifies “extraction rules” that are applied to data in theevents to extract values for specific fields. More specifically, theextraction rules for a field can include one or more instructions thatspecify how to extract a value for the field from the event data. Anextraction rule can generally include any type of instruction forextracting values from data in events. In some cases, an extraction rulecomprises a regular expression, in which case the rule is referred to asa “regex rule.”

In contrast to a conventional schema for a database system, alate-binding schema is not defined at data ingestion time. Instead, thelate-binding schema can be developed on an ongoing basis until the timea query is actually executed. This means that extraction rules for thefields in a query may be provided in the query itself, or may be locatedduring execution of the query. Hence, as an analyst learns more aboutthe data in the events, the analyst can continue to refine thelate-binding schema by adding new fields, deleting fields, or changingthe field extraction rules until the next time the schema is used by aquery. Because the SPLUNK® ENTERPRISE system maintains the underlyingraw data and provides a late-binding schema for searching the raw data,it enables an analyst to investigate questions that arise as the analystlearns more about the events.

In the SPLUNK® ENTERPRISE system, a field extractor may be configured toautomatically generate extraction rules for certain fields in the eventswhen the events are being created, indexed, or stored, or possibly at alater time. Alternatively, a user may manually define extraction rulesfor fields using a variety of techniques.

Also, a number of “default fields” that specify metadata about theevents rather than data in the events themselves can be createdautomatically. For example, such default fields can specify: a timestampfor the event data; a host from which the event data originated; asource of the event data; and a source type for the event data. Thesedefault fields may be determined automatically when the events arecreated, indexed or stored.

In some embodiments, a common field name may be used to reference two ormore fields containing equivalent data items, even though the fields maybe associated with different types of events that possibly havedifferent data formats and different extraction rules. By enabling acommon field name to be used to identify equivalent fields fromdifferent types of events generated by different data sources, thesystem facilitates use of a “common information model” (CIM) across thedifferent data sources.

1.2 Data Server System

FIG. 1 presents a block diagram of an exemplary event-processing system100, similar to the SPLUNK® ENTERPRISE system. System 100 includes oneor more forwarders 101 that collect data obtained from a variety ofdifferent data sources 105, and one or more indexers 102 that store,process, and/or perform operations on this data, wherein each indexeroperates on data contained in a specific data store 103. Theseforwarders and indexers can comprise separate computer systems in a datacenter, or may alternatively comprise separate processes executing onvarious computer systems in a data center.

During operation, the forwarders 101 identify which indexers 102 willreceive the collected data and then forward the data to the identifiedindexers. Forwarders 101 can also perform operations to strip outextraneous data and detect timestamps in the data. The forwarders nextdetermine which indexers 102 will receive each data item and thenforward the data items to the determined indexers 102.

Note that distributing data across different indexers facilitatesparallel processing. This parallel processing can take place at dataingestion time, because multiple indexers can process the incoming datain parallel. The parallel processing can also take place at search time,because multiple indexers can search through the data in parallel.

System 100 and the processes described below with respect to FIGS. 1-5are further described in “Exploring Splunk Search Processing Language(SPL) Primer and Cookbook” by David Carasso, CITO Research, 2012, and in“Optimizing Data Analysis With a Semi-Structured Time Series Database”by Ledion Bitincka, Archana Ganapathi, Stephen Sorkin, and Steve Zhang,SLAML, 2010, each of which is hereby incorporated herein by reference inits entirety for all purposes.

1.3 Data Ingestion

FIG. 2 presents a flow chart illustrating how an indexer processes,indexes, and stores data received from forwarders in accordance with thedisclosed embodiments. At block 201, the indexer receives the data fromthe forwarder. Next, at block 202, the indexer apportions the data intoevents. Note that the data can include lines of text that are separatedby carriage returns or line breaks and an event may include one or moreof these lines. During the apportioning process, the indexer can useheuristic rules to automatically determine the boundaries of the events,which for example coincide with line boundaries. These heuristic rulesmay be determined based on the source of the data, wherein the indexercan be explicitly informed about the source of the data or can infer thesource of the data by examining the data. These heuristic rules caninclude regular expression-based rules or delimiter-based rules fordetermining event boundaries, wherein the event boundaries may beindicated by predefined characters or character strings. Thesepredefined characters may include punctuation marks or other specialcharacters including, for example, carriage returns, tabs, spaces orline breaks. In some cases, a user can fine-tune or configure the rulesthat the indexers use to determine event boundaries in order to adaptthe rules to the user's specific requirements.

Next, the indexer determines a timestamp for each event at block 203. Asmentioned above, these timestamps can be determined by extracting thetime directly from data in the event, or by interpolating the time basedon timestamps from temporally proximate events. In some cases, atimestamp can be determined based on the time the data was received orgenerated. The indexer subsequently associates the determined timestampwith each event at block 204, for example by storing the timestamp asmetadata for each event.

Then, the system can apply transformations to data to be included inevents at block 205. For log data, such transformations can includeremoving a portion of an event (e.g., a portion used to define eventboundaries, extraneous text, characters, etc.) or removing redundantportions of an event. Note that a user can specify portions to beremoved using a regular expression or any other possible technique.

Next, a keyword index can optionally be generated to facilitate fastkeyword searching for events. To build a keyword index, the indexerfirst identifies a set of keywords in block 206. Then, at block 207 theindexer includes the identified keywords in an index, which associateseach stored keyword with references to events containing that keyword(or to locations within events where that keyword is located). When anindexer subsequently receives a keyword-based query, the indexer canaccess the keyword index to quickly identify events containing thekeyword.

In some embodiments, the keyword index may include entries forname-value pairs found in events, wherein a name-value pair can includea pair of keywords connected by a symbol, such as an equals sign orcolon. In this way, events containing these name-value pairs can bequickly located. In some embodiments, fields can automatically begenerated for some or all of the name-value pairs at the time ofindexing. For example, if the string “dest=10.0.1.2” is found in anevent, a field named “dest” may be created for the event, and assigned avalue of “10.0.1.2.”

Finally, the indexer stores the events in a data store at block 208,wherein a timestamp can be stored with each event to facilitatesearching for events based on a time range. In some cases, the storedevents are organized into a plurality of buckets, wherein each bucketstores events associated with a specific time range. This not onlyimproves time-based searches, but it also allows events with recenttimestamps that may have a higher likelihood of being accessed to bestored in faster memory to facilitate faster retrieval. For example, abucket containing the most recent events can be stored as flash memoryinstead of on hard disk.

Each indexer 102 is responsible for storing and searching a subset ofthe events contained in a corresponding data store 103. By distributingevents among the indexers and data stores, the indexers can analyzeevents for a query in parallel, for example using map-reduce techniques,wherein each indexer returns partial responses for a subset of events toa search head that combines the results to produce an answer for thequery. By storing events in buckets for specific time ranges, an indexermay further optimize searching by looking only in buckets for timeranges that are relevant to a query.

Moreover, events and buckets can also be replicated across differentindexers and data stores to facilitate high availability and disasterrecovery as is described in U.S. patent application Ser. No. 14/266,812filed on 30 Apr. 2014, and in U.S. patent application Ser. No.14/266,817 also filed on 30 Apr. 2014.

1.4 Query Processing

FIG. 3 presents a flow chart illustrating how a search head and indexersperform a search query in accordance with the disclosed embodiments. Atthe start of this process, a search head receives a search query from aclient at block 301. Next, at block 302, the search head analyzes thesearch query to determine what portions can be delegated to indexers andwhat portions need to be executed locally by the search head. At block303, the search head distributes the determined portions of the query tothe indexers. Note that commands that operate on single events can betrivially delegated to the indexers, while commands that involve eventsfrom multiple indexers are harder to delegate.

Then, at block 304, the indexers to which the query was distributedsearch their data stores for events that are responsive to the query. Todetermine which events are responsive to the query, the indexer searchesfor events that match the criteria specified in the query. This criteriacan include matching keywords or specific values for certain fields. Ina query that uses a late-binding schema, the searching operations inblock 304 may involve using the late-binding scheme to extract valuesfor specified fields from events at the time the query is processed.Next, the indexers can either send the relevant events back to thesearch head, or use the events to calculate a partial result, and sendthe partial result back to the search head.

Finally, at block 305, the search head combines the partial resultsand/or events received from the indexers to produce a final result forthe query. This final result can comprise different types of datadepending upon what the query is asking for. For example, the finalresults can include a listing of matching events returned by the query,or some type of visualization of data from the returned events. Inanother example, the final result can include one or more calculatedvalues derived from the matching events.

Moreover, the results generated by system 100 can be returned to aclient using different techniques. For example, one technique streamsresults back to a client in real-time as they are identified. Anothertechnique waits to report results to the client until a complete set ofresults is ready to return to the client. Yet another technique streamsinterim results back to the client in real-time until a complete set ofresults is ready, and then returns the complete set of results to theclient. In another technique, certain results are stored as “searchjobs,” and the client may subsequently retrieve the results byreferencing the search jobs.

The search head can also perform various operations to make the searchmore efficient. For example, before the search head starts executing aquery, the search head can determine a time range for the query and aset of common keywords that all matching events must include. Next, thesearch head can use these parameters to query the indexers to obtain asuperset of the eventual results. Then, during a filtering stage, thesearch head can perform field-extraction operations on the superset toproduce a reduced set of search results.

1.5 Field Extraction

FIG. 4 presents a block diagram illustrating how fields can be extractedduring query processing in accordance with the disclosed embodiments. Atthe start of this process, a search query 402 is received at a queryprocessor 404. Query processor 404 includes various mechanisms forprocessing a query, wherein these mechanisms can reside in a search head104 and/or an indexer 102. Note that the exemplary search query 402illustrated in FIG. 4 is expressed in the Search Processing Language(SPL), which is used in conjunction with the SPLUNK® ENTERPRISE system.SPL is a pipelined search language in which a set of inputs is operatedon by a first command in a command line, and then a subsequent commandfollowing the pipe symbol “I” operates on the results produced by thefirst command, and so on for additional commands. Search query 402 canalso be expressed in other query languages, such as the Structured QueryLanguage (“SQL”) or any suitable query language.

Upon receiving search query 402, query processor 404 sees that searchquery 402 includes two fields “IP” and “target.” Query processor 404also determines that the values for the “IP” and “target” fields havenot already been extracted from events in data store 414, andconsequently determines that query processor 404 needs to use extractionrules to extract values for the fields. Hence, query processor 404performs a lookup for the extraction rules in a rule base 406, whereinrule base 406 maps field names to corresponding extraction rules andobtains extraction rules 408-409, wherein extraction rule 408 specifieshow to extract a value for the “IP” field from an event, and extractionrule 409 specifies how to extract a value for the “target” field from anevent. As is illustrated in FIG. 4, extraction rules 408-409 cancomprise regular expressions that specify how to extract values for therelevant fields. Such regular-expression-based extraction rules are alsoreferred to as “regex rules.” In addition to specifying how to extractfield values, the extraction rules may also include instructions forderiving a field value by performing a function on a character string orvalue retrieved by the extraction rule. For example, a transformationrule may truncate a character string, or convert the character stringinto a different data format. In some cases, the query itself canspecify one or more extraction rules.

Next, query processor 404 sends extraction rules 408-409 to a fieldextractor 412, which applies extraction rules 408-409 to events 416-418in a data store 414. Note that data store 414 can include one or moredata stores, and extraction rules 408-409 can be applied to largenumbers of events in data store 414, and are not meant to be limited tothe three events 416-418 illustrated in FIG. 4. Moreover, the queryprocessor 404 can instruct field extractor 412 to apply the extractionrules to all the events in a data store 414, or to a subset of theevents that has been filtered based on some criteria.

Next, field extractor 412 applies extraction rule 408 for the firstcommand “Search IP=“10*” to events in data store 414 including events416-418. Extraction rule 408 is used to extract values for the IPaddress field from events in data store 414 by looking for a pattern ofone or more digits, followed by a period, followed again by one or moredigits, followed by another period, followed again by one or moredigits, followed by another period, and followed again by one or moredigits. Next, field extractor 412 returns field values 420 to queryprocessor 404, which uses the criterion IP=“10*” to look for IPaddresses that start with “10”. Note that events 416 and 417 match thiscriterion, but event 418 does not, so the result set for the firstcommand is events 416-417.

Query processor 404 then sends events 416-417 to the next command “statscount target.” To process this command, query processor 404 causes fieldextractor 412 to apply extraction rule 409 to events 416-417. Extractionrule 409 is used to extract values for the target field for events416-417 by skipping the first four commas in events 416-417, and thenextracting all of the following characters until a comma or period isreached. Next, field extractor 412 returns field values 421 to queryprocessor 404, which executes the command “stats count target” to countthe number of unique values contained in the target fields, which inthis example produces the value “2” that is returned as a final result422 for the query.

Note that query results can be returned to a client, a search head, orany other system component for further processing. In general, queryresults may include: a set of one or more events; a set of one or morevalues obtained from the events; a subset of the values; statisticscalculated based on the values; a report containing the values; or avisualization, such as a graph or chart, generated from the values.

1.6 Exemplary Search Screen

FIG. 6A illustrates an exemplary search screen 600 in accordance withthe disclosed embodiments. Search screen 600 includes a search bar 602that accepts user input in the form of a search string. It also includesa time range picker 612 that enables the user to specify a time rangefor the search. For “historical searches” the user can select a specifictime range, or alternatively a relative time range, such as “today,”“yesterday” or “last week.” For “real-time searches,” the user canselect the size of a preceding time window to search for real-timeevents. Search screen 600 also initially displays a “data summary”dialog as is illustrated in FIG. 6B that enables the user to selectdifferent sources for the event data, for example by selecting specifichosts and log files.

After the search is executed, the search screen 600 can display theresults through search results tabs 604, wherein search results tabs 604includes: an “events tab” that displays various information about eventsreturned by the search; a “statistics tab” that displays statisticsabout the search results; and a “visualization tab” that displaysvarious visualizations of the search results. The events tab illustratedin FIG. 6A displays a timeline graph 605 that graphically illustratesthe number of events that occurred in one-hour intervals over theselected time range. It also displays an events list 608 that enables auser to view the raw data in each of the returned events. Itadditionally displays a fields sidebar 606 that includes statisticsabout occurrences of specific fields in the returned events, including“selected fields” that are pre-selected by the user, and “interestingfields” that are automatically selected by the system based onpre-specified criteria.

1.7 Acceleration Techniques

The above-described system provides significant flexibility by enablinga user to analyze massive quantities of minimally processed performancedata “on the fly” at search time instead of storing pre-specifiedportions of the performance data in a database at ingestion time. Thisflexibility enables a user to see correlations in the performance dataand perform subsequent queries to examine interesting aspects of theperformance data that may not have been apparent at ingestion time.

However, performing extraction and analysis operations at search timecan involve a large amount of data and require a large number ofcomputational operations, which can cause considerable delays whileprocessing the queries. Fortunately, a number of acceleration techniqueshave been developed to speed up analysis operations performed at searchtime. These techniques include: (1) performing search operations inparallel by formulating a search as a map-reduce computation; (2) usinga keyword index; (3) using a high performance analytics store; and (4)accelerating the process of generating reports. These techniques aredescribed in more detail below.

1.7.1 Map-Reduce Technique

To facilitate faster query processing, a query can be structured as amap-reduce computation, wherein the “map” operations are delegated tothe indexers, while the corresponding “reduce” operations are performedlocally at the search head. For example, FIG. 5 illustrates how a searchquery 501 received from a client at search head 104 can split into twophases, including: (1) a “map phase” comprising subtasks 502 (e.g., dataretrieval or simple filtering) that may be performed in parallel and are“mapped” to indexers 102 for execution, and (2) a “reduce phase”comprising merging operations 503 to be executed by the search head whenthe results are ultimately collected from the indexers.

During operation, upon receiving search query 501, search head 104modifies search query 501 by substituting “stats” with “prestats” toproduce search query 501, and then distributes search query 501 to oneor more distributed indexers, which are also referred to as “searchpeers.” Note that search queries may generally specify search criteriaor operations to be performed on events that meet the search criteria.Search queries may also specify field names, as well as search criteriafor the values in the fields or operations to be performed on the valuesin the fields. Moreover, the search head may distribute the full searchquery to the search peers as is illustrated in FIG. 3, or mayalternatively distribute a modified version (e.g., a more restrictedversion) of the search query to the search peers. In this example, theindexers are responsible for producing the results and sending them tothe search head. After the indexers return the results to the searchhead, the search head performs the merging operations 503 on theresults. Note that by executing the computation in this way, the systemeffectively distributes the computational operations while minimizingdata transfers.

1.7.2 Keyword Index

As described above with reference to the flow charts in FIGS. 2 and 3,event-processing system 100 can construct and maintain one or morekeyword indices to facilitate rapidly identifying events containingspecific keywords. This can greatly speed up the processing of queriesinvolving specific keywords. As mentioned above, to build a keywordindex, an indexer first identifies a set of keywords. Then, the indexerincludes the identified keywords in an index, which associates eachstored keyword with references to events containing that keyword, or tolocations within events where that keyword is located. When an indexersubsequently receives a keyword-based query, the indexer can access thekeyword index to quickly identify events containing the keyword.

1.7.3 High Performance Analytics Store

To speed up certain types of queries, some embodiments of system 100make use of a high performance analytics store, which is referred to asa “summarization table,” that contains entries for specific field-valuepairs. Each of these entries keeps track of instances of a specificvalue in a specific field in the event data and includes references toevents containing the specific value in the specific field. For example,an exemplary entry in a summarization table can keep track ofoccurrences of the value “94107” in a “ZIP code” field of a set ofevents, wherein the entry includes references to all of the events thatcontain the value “94107” in the ZIP code field. This enables the systemto quickly process queries that seek to determine how many events have aparticular value for a particular field, because the system can examinethe entry in the summarization table to count instances of the specificvalue in the field without having to go through the individual events ordo extractions at search time. Also, if the system needs to process allevents that have a specific field-value combination, the system can usethe references in the summarization table entry to directly access theevents to extract further information without having to search all ofthe events to find the specific field-value combination at search time.

In some embodiments, the system maintains a separate summarization tablefor each of the above-described time-specific buckets that stores eventsfor a specific time range, wherein a bucket-specific summarization tableincludes entries for specific field-value combinations that occur inevents in the specific bucket. Alternatively, the system can maintain aseparate summarization table for each indexer, wherein theindexer-specific summarization table only includes entries for theevents in a data store that is managed by the specific indexer.

The summarization table can be populated by running a “collection query”that scans a set of events to find instances of a specific field-valuecombination, or alternatively instances of all field-value combinationsfor a specific field. A collection query can be initiated by a user, orcan be scheduled to occur automatically at specific time intervals. Acollection query can also be automatically launched in response to aquery that asks for a specific field-value combination.

In some cases, the summarization tables may not cover all of the eventsthat are relevant to a query. In this case, the system can use thesummarization tables to obtain partial results for the events that arecovered by summarization tables, but may also have to search throughother events that are not covered by the summarization tables to produceadditional results. These additional results can then be combined withthe partial results to produce a final set of results for the query.This summarization table and associated techniques are described in moredetail in U.S. Pat. No. 8,682,925, issued on Mar. 25, 2014.

1.7.4 Accelerating Report Generation

In some embodiments, a data server system such as the SPLUNK® ENTERPRISEsystem can accelerate the process of periodically generating updatedreports based on query results. To accelerate this process, asummarization engine automatically examines the query to determinewhether generation of updated reports can be accelerated by creatingintermediate summaries. (This is possible if results from preceding timeperiods can be computed separately and combined to generate an updatedreport. In some cases, it is not possible to combine such incrementalresults, for example where a value in the report depends onrelationships between events from different time periods.) If reportscan be accelerated, the summarization engine periodically generates asummary covering data obtained during a latest non-overlapping timeperiod. For example, where the query seeks events meeting a specifiedcriteria, a summary for the time period includes only events within thetime period that meet the specified criteria. Similarly, if the queryseeks statistics calculated from the events, such as the number ofevents that match the specified criteria, then the summary for the timeperiod includes the number of events in the period that match thespecified criteria.

In parallel with the creation of the summaries, the summarization engineschedules the periodic updating of the report associated with the query.During each scheduled report update, the query engine determines whetherintermediate summaries have been generated covering portions of the timeperiod covered by the report update. If so, then the report is generatedbased on the information contained in the summaries. Also, if additionalevent data has been received and has not yet been summarized, and isrequired to generate the complete report, the query can be run on thisadditional event data. Then, the results returned by this query on theadditional event data, along with the partial results obtained from theintermediate summaries, can be combined to generate the updated report.This process is repeated each time the report is updated. Alternatively,if the system stores events in buckets covering specific time ranges,then the summaries can be generated on a bucket-by-bucket basis. Notethat producing intermediate summaries can save the work involved inre-running the query for previous time periods, so only the newer eventdata needs to be processed while generating an updated report. Thesereport acceleration techniques are described in more detail in U.S. Pat.No. 8,589,403, issued on Nov. 19, 2013, and U.S. Pat. No. 8,412,696,issued on Apr. 2, 2011.

System for Generating Alerts

FIG. 7 illustrates a system 700 for generating alerts based on searchresults produced by a query along with associated metadata associated inaccordance with the disclosed embodiments. The system includes a queryprocessor 404 that execute queries on behalf of a user 702 who inputsthe queries through a search UI 704. These queries can operate on eventsobtained from data store 103 to produce search results and associatedmetadata 708. Note that this associated metadata can include metadataabout the search itself, such as the name of the search and the runningtime of the search. Note that the operations performed by queryprocessor 404 are described in more detail above with reference to FIG.4.

Search results and associated metadata 708 feed into an alert generator710, which can generate an alert based on the search results andassociated metadata 708. This alert can be configured by a user 712 whocan input various parameters for the alert through an alert UI 714. Notethat user 712 and user 702 can possibly be the same user. Also note thatalert UI 714 can be rendered based on UI code 713, which is receivedfrom a third-party developer who is not part of an organization thatowns system 700. This enables the third-party developer to generate acustom alert UI, which is customized to generate specific alert actionsas is described in more detail below with reference to FIG. 10. Duringoperation of the system illustrated in FIG. 7, the parameters entered byuser 712 through alert UI 714 are used to generate alert configurationinformation 716, which feeds into alert generator 710.

Alert generator 710 uses this alert configuration information 716 togenerate alerts based on search results and associated metadata 708.During this process, alert generator 710 assembles a payload 718containing parameters for the alert and then triggers execution of ascript 720 (or application) that uses the parameters in payload 718 toperform one or more alert actions 722 for each query result. This canhappen in a number of ways. A query result can trigger a single alertaction that sends out multiple alerts. For example, a single alertaction might generate multiple ServiceNow™ tickets. Or, a query resultcan trigger multiple alert actions, such as separate alert actions thatgenerate a ServiceNow™ ticket and that send a HipChat message.

In some embodiments, a “spec file” can be used to define the inputparameters used by the script. Note that during development of UI code713, the programmer can bind input parameters defined in the script touser-interface fields to enable the user to enter the input parameters.

Payloads

FIGS. 8A and 8B illustrate exemplary payloads containing parameters foralert actions in accordance with the disclosed embodiments. Morespecifically, FIG. 8A illustrates an exemplary payload expressed ineXtensible Markup Language (XML) format, and FIG. 8B illustrates anexemplary payload expressed in JavasScript Object Notation (JSON)format. Referring to FIG. 8B, the first few parameters “server_host”:“localhost:8089”, “server_url”: “https://localhost:8089”, “session_key”:“1234512345” provide information specifying how to communicate with aserver that operates query processor 404. The following parametersspecify a location where the results are stored.

“/opt/splunk/var/run/splunk/12938718293123.121/results.csv.gz”

“results_link”:

“http://splunk.server.local:8000/en-US/app/search?sid=12341234.123”,

This provides a user with a “deep link” into the search results, whichenables the user to easily look at specific details of the searchresults. The payload illustrated in FIG. 8B also includes parametersassociated with the search that appear below.

“sid”: “12341234.123”,

“search_name”: “My Saved Search”,

“owner”: “admin”,

“app”: “search”,

These parameters include the search ID, the name of the search and theowner/application associated with the search. These parameters can beused to facilitate communications with the server. Finally, the payloadincludes various search-related parameters, including metadataparameters associated with the search and data parameters obtained fromfields in the search results as is described in more detail below.Alert-Generation UI

FIG. 9 illustrates an exemplary alert-generation user interface (UI) 900in accordance with the disclosed embodiments. Exemplary alert-generationUI 900 enables a user to input a title 902 and a description 904 for thealert. It also enables the user to set permissions 906 for the alert.For example, as illustrated in FIG. 9, the alert may be “private” or“shared.” Permissions may also be set to make an alert action availablefor only a specific application, or for only a specific subset of users.An alert action can also be selectively enabled and disabled. Forexample, an alert action may be disabled for a month for a specificgroup of users. The user can also specify an alert type 908 thatindicates whether the alert is a real-time alert, which takes place uponoccurrence of a specific event, or is a scheduled alert that runs, forexample, every Monday at 6:00 AM. Alert-generation UI 900 also enablesthe user to specify various trigger conditions 910 for triggering analert. For example, the alert can be triggered when the number ofresults is greater than a value such as five. Moreover, the alert can betriggered once for all five results to facilitate batch processing, orit can be triggered once for each result. The user can also specify oneor more alert actions 912, such as sending an email or running aspecific application.

Note that the system also enables a user to install a customalert-action (written by a third-party developer) into the system. Forexample, FIG. 10 illustrates an exemplary custom alert-action UI 1000for the “HipChat” application in accordance with the disclosedembodiments. In this example, when the HipChat application is triggeredas an alert action, it receives a number of parameters from customalert-action UI 1000, including: (1) the name of a HipChat room 1002,(2) a chat message 1004 to be sent to the HipChat room, (3) a messageformat 1006 indicating whether the message is in plain text format orHTML format, (4) a message color 1008 for the message, (5) an indication1010 of whether to notify the other users in the room, and (6) anoptional authentication token 1012 that can be used to override aglobally configured authentication token for the room. This customalert-action UI 1000 can be generated by a third-party developer (who,for example, writes XML or HTML code), and this custom alert-action UI1000 can be integrated into the alert generation system illustrated inFIG. 7.

Note that enabling the alert action to operate on pre-specifiedparameters associated with query result is powerful. For example, in acomputer-security use-case, the chat message 1004 described above caninclude information identifying an IP address that is sendingproblematic emails. In this way, a system administrator who views thechat message can take actions to block emails from the problematic IPaddress. Also note that the developer creating the alert action can askto receive specific parameters associated with the query associatedsearch results by specifying the parameters in a “spec file” asdescribed in more detail below.

Moreover, note that a normal user would not be expected to know what isrequired to invoke a specific alert action. What makes this techniquepowerful is that a developer, who writes the UI for configuring thealert action, can define fields for the required parameters based on theabove-described “spec file.” This enables the user to enter the requiredparameters for the alert action through the UI, without having to knowwhat parameters are required for the alert action beforehand.

Also note that the chat message entered into the “message” fieldincludes a number of tokens, including, “$result.user,”“$result.reason$,” and “$result.clientip$.” Before this message field issent to the HipChat application, the system substitutes correspondingparameters from the search results and associated metadata in place ofthese tokens. In this way, the message sent to the HipChat roomidentifies: (1) a specific user associated with the failed loginattempts, (2) a reason for the failed login attempts, and (3) the clientIP address associated with the failed login attempts. Note that theseparameters can be obtained from the search results and associatedmetadata by using a late-binding schema to extract the parameters fromthe search results and associated events. Also note that because thetoken substitutions are performed by the system prior to invoking thescript, while the system is generating the payload, the script does notneed to handle any of the details of the token substitution. The scriptdeveloper simply defines the input parameters for the script in the specfile (as described above), and the developer for the correspondingalert-action UI incorporates fields for these input parameters into thealert-action UI.

A number of different types of parameters associated with a search canbe tokenized. For example, the system can tokenize metadata about thesearch, such as the search name and the owner of the search. The systemcan also tokenize metadata about the process that executes the search,such as the time the search was executed and the running time of thesearch. The system can additionally tokenize metadata about the serverthat executed the search, such as the IP address of the server and thename of the server. Finally, the system can tokenize one or moreparameters in the search results and associated metadata. This meansthat parameters obtained from the search results and associated metadatacan be incorporated into the payload that is used to generate the alertaction.

FIG. 11 illustrates an exemplary email alert-action UI 1100 inaccordance with the disclosed embodiments. Email alert-action UI 1100enables the user to enter parameters that specify how an email is sentwhile processing an alert. More specifically, email alert-action UI 1100enables the user to enter: (1) one or more email addresses in field 1102for one or more email recipients, (2) a priority for the email 1104, (3)a subject line for the email 1106, (4) a message to be included in thebody of the email 1108, (5) checkboxes 1110 for various items to includewith the email, and (6) a specifier 1112 for whether the email is “HTML& Plain Text,” or just “Plain Text.” Unlike custom alert-action UI 1000illustrated in FIG. 10, email alert-action UI 1100 can possibly be an“out-of-the-box” alert-action UI that comprises a standard feature ofthe system 700 illustrated in FIG. 7.

Also note that both the subject field and the message field of emailalert-action UI 1100 include the token “$name$.” Hence, before thestrings in the subject field and the message field are sent to the emailapplication, the name of the specific alert that triggered the email issubstituted in place of the token “$name$.” This causes the resultingemail to identify the specific alert.

Process of Generating Alerts

FIG. 12A presents a flow chart illustrating how alerts are generated inaccordance with the disclosed embodiments. At the start of this process,the system executes a query to generate the search results (step 1202).Next, the system obtains configuration information for the alert,wherein the configuration information identifies information to beextracted from the search results along with metadata associated withthe search results, and also specifies a trigger condition for the alert(step 1204).

Then, the system monitors the trigger condition for the alert todetermine when the trigger condition is met (step 1206). Finally, whenthe trigger condition for the alert is met, the system performs a numberof operations. First, the system uses the configuration information togenerate a payload containing the identified information from the searchresults and associated metadata (step 1208). Next, the system invokesthe alert-generating functionality (step 1210). For example, the systemcan spawn a process that executes a script containing thealert-generating functionality. Finally, the system provides the payloadas input to the alert-generating functionality, which enables thealert-generating functionality to use the information from the searchresults and associated metadata while generating the alert (step 1212).

Ad Hoc Alerts

Some embodiments of the present invention also provide support for “adhoc alerts” that can be manually triggered for a specific result set (incontrast to conventional alerts that are automatically triggered basedon pre-specified trigger conditions). For example, referring the FIG.12B, a user can execute a query to generate search results andassociated metadata (step 112), and can then manually trigger an alertbased on the search results and associated metadata (step 1220). Forexample, FIG. 12C presents a screenshot of a UI 1250 illustrating how analert action can be manually triggered by entering the text string “|sendmodalert” 1252 into the new search bar 1254.

FIG. 12D presents another example illustrating how an alert action canbe manually triggered through a UI 1260. In this example, a user who isviewing UI 1260, which is associated with a specific event, can activatea pull-down menu 1262 that contains a list of possible event actions.One of these event actions is an “alert action” that sends a HipChatnotification 1264. (Recall that this HipChat alert action was previouslydiscussed with reference to UI 1000 illustrated in FIG. 10.) Byselecting this HipChat notification 1264 from pull-down menu 1262, theuser can manually trigger the HipChat notification 1264 for the specificevent. Note that this Hipchat notification 1264 was previouslyconfigured by the user through UI 1000, so the user does not haveconfigure it again; the user simply triggers this pre-configured alertaction. In an alternative embodiment, if the alert action that wasselected to be manually triggered has not been pre-configured yet, orneeds to be reconfigured, the user can be presented with a UI 1000 thatenables the user to configure or reconfigure the alert action.

The above-described ad hoc alert functionality enables a user or anadministrator to manually invoke an alert action “on demand” for testingpurposes or other purposes. For example, in a computer-security usecase, when a network administrator determines that packets received froma specific IP address are problematic, the network administrator canmanually trigger an alert action that modifies a firewall rule, andwhile doing so, can pass the problematic IP address as a parameter tothe alert action. This enables the firewall to block the problematic IPaddress.

Another type of ad hoc alert is an “event-generated alert” that is notnecessarily associated with search results generated by a query. Forexample, a user may be viewing a collection of events on a screen, andmay expand a drop-down on one of the events that offers options foroperations that can be performed on the event. One of these operationscan comprise an alert action to be applied to the event, such asgenerating a ServiceNow™ ticket for the event.

Process of Configuring an Alert Action

FIG. 13 presents a flow chart illustrating how a user configures analert action in accordance with the disclosed embodiments. At the startof this process, the system presents an alert user interface (UI) to auser, wherein the alert UI enables the user to configure one or morealert actions to be performed based on the search results and associatedmetadata (step 1302). Next, the system receives alert configurationinformation from the user through the alert UI, wherein the alertconfiguration information includes tokens representing parametersassociated with the query and the search results (step 1304). Finally,while generating an alert associated with the search results, the systemperforms a token substitution for the tokens in the alert configurationinformation to generate a payload that is communicated toalert-generating functionality, so that the parameters can be used bythe alert-generating functionality while performing the one or morealert actions (step 1306).

The preceding description was presented to enable any person skilled inthe art to make and use the disclosed embodiments, and is provided inthe context of a particular application and its requirements. Variousmodifications to the disclosed embodiments will be readily apparent tothose skilled in the art, and the general principles defined herein maybe applied to other embodiments and applications without departing fromthe spirit and scope of the disclosed embodiments. Thus, the disclosedembodiments are not limited to the embodiments shown, but are to beaccorded the widest scope consistent with the principles and featuresdisclosed herein. Accordingly, many modifications and variations will beapparent to practitioners skilled in the art. Additionally, the abovedisclosure is not intended to limit the present description. The scopeof the present description is defined by the appended claims.

The data structures and code described in this detailed description aretypically stored on a computer-readable storage medium, which may be anydevice or medium that can store code and/or data for use by a system.The computer-readable storage medium includes, but is not limited to,volatile memory, non-volatile memory, magnetic and optical storagedevices such as disk drives, magnetic tape, CDs (compact discs), DVDs(digital versatile discs or digital video discs), or other media capableof storing code and/or data now known or later developed.

The methods and processes described in the detailed description sectioncan be embodied as code and/or data, which can be stored on anon-transitory computer-readable storage medium as described above. Whena system reads and executes the code and/or data stored on thenon-transitory computer-readable storage medium, the system performs themethods and processes embodied as data structures and code and storedwithin the non-transitory computer-readable storage medium.

Furthermore, the methods and processes described above can be includedin hardware modules. For example, the hardware modules can include, butare not limited to, application-specific integrated circuit (ASIC)chips, field-programmable gate arrays (FPGAs), and otherprogrammable-logic devices now known or later developed. When thehardware modules are activated, the hardware modules perform the methodsand processes included within the hardware modules.

What is claimed is:
 1. A method, comprising: periodically executing aquery to generate a set of search results; based on the set of searchresults, retrieving configuration information for an alert, theconfiguration information specifies parameters for a trigger conditionfor the alert, the alert is an indication of activity in an informationtechnology environment; extracting information from the set of searchresults upon satisfying the parameters for the trigger condition for thealert; substituting a token in a string with at least a portion of theextracted information; incorporating the string into an alert input; andcausing an alert-generating functionality to perform one or more alertactions using the alert input.
 2. The method of claim 1, furthercomprising: allowing a user to manually invoke an alert action based onthe set of search results.
 3. The method of claim 1, wherein theextracting information from the set of search results further comprises:using a late-binding schema to extract the information from the set ofsearch results.
 4. The method of claim 1, wherein the extractinginformation from the set of search results further comprises: using alate-binding schema to extract the information from the set of searchresults; using the extracted information to perform one or more tokensubstitutions in strings that are incorporated into the alert input. 5.The method of claim 1, wherein the configuration information is definedthrough a user interface by a user, and wherein the configurationinformation specifies how the extracted information is used to performthe one or more alert actions.
 6. The method of claim 1, wherein the oneor more alert actions includes sending a chat message to a chat room. 7.The method of claim 1, further comprising: wherein the one or more alertactions includes sending a chat message to a chat room; performing oneor more token substitutions in strings, using the extracted information,that are incorporated into the chat message.
 8. The method of claim 1,wherein the one or more alert actions includes sending a notification toone or more users.
 9. The method of claim 1, further comprising: whereinthe one or more alert actions includes sending a notification to one ormore users; performing one or more token substitutions in strings, usingthe extracted information, that are incorporated into the notification.10. The method of claim 1, wherein the configuration informationincludes one or more of: a title of the alert; a description of thealert; an indicator for when the alert is scheduled to be invoked; oneor more permissions associated with the alert; one or more triggerconditions for the alert; or the one or more alert actions.
 11. Anon-transitory computer-readable storage medium storing instructionsthat when executed by a computer cause the computer to perform a methodfor generating an alert based on information associated with searchresults generated by a query, the method comprising: periodicallyexecuting a query to generate a set of search results; based on the setof search results, retrieving configuration information for an alert,the configuration information specifies parameters for a triggercondition for the alert, the alert is an indication of activity in aninformation technology environment; extracting information from the setof search results upon satisfying the parameters for the triggercondition for the alert; substituting a token in a string with at leasta portion of the extracted information; incorporating the string into analert input; and causing an alert-generating functionality to performone or more alert actions using the alert input.
 12. The non-transitorycomputer-readable storage medium of claim 11, wherein the extractinginformation from the set of search results further comprises: using alate-binding schema to extract the information from the set of searchresults.
 13. The non-transitory computer-readable storage medium ofclaim 11, wherein the extracting information from the set of searchresults further comprises: using a late-binding schema to extract theinformation from the set of search results; using the extractedinformation to perform one or more token substitutions in strings thatare incorporated into the alert input.
 14. The non-transitorycomputer-readable storage medium of claim 11, wherein the configurationinformation is defined through a user interface by a user, and whereinthe configuration information specifies how the extracted information isused to perform the one or more alert actions.
 15. The non-transitorycomputer-readable storage medium of claim 11, wherein the one or morealert actions includes sending a chat message to a chat room.
 16. Anapparatus, comprising: one or more processors; and a memory storinginstructions, which when executed by the one or more processors, causesthe one or more processors to: periodically execute a query to generatea set of search results; based on the set of search results, retrieveconfiguration information for an alert, the configuration informationspecifies parameters for a trigger condition for the alert, the alert isan indication of activity in an information technology environment;extract information from the set of search results upon satisfying theparameters for the trigger condition for the alert; substitute a tokenin a string with at least a portion of the extracted information;incorporate the string into an alert input; and cause analert-generating functionality to perform one or more alert actionsusing the alert input.
 17. The apparatus of claim 16, wherein theextract information from the set of search results further comprises:use a late-binding schema to extract the information from the set ofsearch results.
 18. The apparatus of claim 16, wherein the extractinformation from the set of search results further comprises: use alate-binding schema to extract the information from the set of searchresults; use the extracted information to perform one or more tokensubstitutions in strings that are incorporated into the alert input. 19.The apparatus of claim 16, wherein the configuration information isdefined through a user interface by a user, and wherein theconfiguration information specifies how the extracted information isused to perform the one or more alert actions.
 20. The apparatus ofclaim 16, wherein the one or more alert actions includes sending a chatmessage to a chat room.